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About this guide 



This guide provides information about: 

• M anually estimating replication performance for one outstanding write 

• Determining the impact of distance on replication performance 

• Using the Performance Estimator spreadsheetto estimate replication performance for one outstanding 
write 

• Using the output from the manual calculation and the spreadsheetto determine the minimum expected 
performance in 1/ 0 operations per second (10 PS) for a particular 1/ 0 size, separation distance, and 
intersite link technology 

• Estimating the ideal maximum throughput requirements for multiple outstanding writes 

Intended audience 

This guide is intended for customers who: 

• Purchased the HP StorageW orks Continuous Access Enterprise Virtual Array (EVA) 

• W ant to estimate the effects of distance on applications that use HP StorageW orks Continuous Access 
EVA 

Prerequisites 

Prerequisites for using this product include knowledge of: 

• SAN fabric configurations 

• Disaster planning 

« HP StorageW orks Enterprise Virtual Array (EVA) 

Related documentation 

In addition to this guide, pleas see other documents for this product: 

• H P StorageW orks Continuous Access EVA administrator guide 

• H P StorageW orks Continuous Access EVA planning guide 

• H P StorageW orks SAN design reference guide 

These and other HP documents can be found on an HP web site: http :/ / www.docs.hp.com . 
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Document conventions and symbols 



Table 1 Document conventions 



Convention 


Element 


M edium blue text: Figure 1 


Cross-reference links and e-mail addresses 


Medium blue, underlined text 
(http:/ / www.hp.com) 


W eb site addresses 


Bold font 


• Key names 

• Text typed into a GUI element such as into a box 

• GUI elements that are clicked or selected, such as menu and list 
items, buttons, and check boxes 


Ita lies fn nt 

1 LU 1 1 V_ -D 1 U II L 


Tpyj- prnnha^ic; 

ILAL V_ 1 1 1 yj 1 1 H -D 1 J> 


Monospace font 


• File and directory names 

• Sv^tpm niitnut 

J y J III \J U VXJ U L 

• Code 

• Texttyped atthe command-line 


Monospace, italic font 


• Code variables 

• Command-line variables 


Monospace, bold font 


Emphasis of file and directory names, system output code, and text 
typed atthe command-line 



WARNING! Indicates thatfailure to follow directions could result in bodily harm or death. 



CAUTION: Indicates thatfailure to follow directions could result in damage to equipment or data. 



Vr)f IMPORTANT: Provides clarifying information or specific instructions. 



r2i^> NOTE: Provides additional information. 



2(V TIP: Provides helpful hints and shortcuts. 
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HP 



technical support 



Telephone numbers for worldwide technical supportare listed on the HP web site: 
http :/ / www.hp.com/ support/ . 

Collect the following information before calling: 

• Technical support registration number (if applicable) 

• Product serial numbers 

• Product model names and numbers 

• Applicable error messages 

• 0 perating system type and revision level 

• Detailed, specific questions 

For continuous quality improvement calls may be recorded or monitored. 

HP strongly recommends that customers sign-up online using the Subscriber's choice web site at 
http:/ / www.hp.com/ go/ e-updates . 

• Subscribing to this service provides you with email updates on the latest product enhancements, newest 
versions of drivers, and firmware documentation updates as well as instant access to numerous other 
product resources. 

• After signing-up, you can quickly locate your products by selecting Business supportand then Storage 
under Product Category. 



For the name of your nearest HP-authorized reseller: 

• In the United States, call 1-800-345-1518. 

• Elsewhere, visit http:/ / www.hp.com and click ContactHP to find locations and telephone numbers. 



For third-party product information, see the following vendor web sites: 

• http:/ / www.hp.com 

• http:/ / www.hp.com/ go/ storage 

• http:/ / www.hp.com/ support/ 

• http:/ / www.docs.hp.com 



HP-authorized reseller 



Helpful web sites 
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1 Introduction 



W hen you are designing an HP StorageW orks Continuous Access solution, it is important to understand 
and plan the distance between the two sites to ensure optimum performance and disaster tolerance. A 
shorter distance between sites will improve performance, but it will also increase the risk of a disaster 
affecting both sites (reduced disaster tolerance). A longer distance between sites will reduce the risk of 
disaster affecting both sites (improved disaster tolerance), but it will also decrease performance. The key to 
a successful solution is creating a balance between performance and protection. 

This chapter describes the replication process and the effect of bandwidth on replication performance. 
Using these two components, you can use this guide to estimate: 

• The performance capabilities of an intersite link technology. 

• The link requirements for a given level of performance. 



Replication process 

To complete one replication 1/ 0 , Continuous Access EVA requires one round trip through the intersite link 
as follows: 

1. The source array sends the data to the destination array. 

2. The destination array receives and stores the data in its local cache and returns an acknowledgement 
to the source array. 



Bandwidth and replication performance 

HP Continuous Access products can move data at extreme distances. However, the inherent propagation 
delays in data transfers can affect the amount of time it takes to complete replication 1/ 0 . At these extreme 
distances, bandwidth is not typically the limiting factor in replication performance; rather it is the delay 
imposed by that distance. At short distances, bandwidth affects performance more than the 
distance-induced delay does. 

To better understand the relationship between bandwidth, distance, and performance, consider a low 
bandwidth intersite link and a high bandwidth intersite link that are both moving 1/ 0 packets containing 
the same amountof data (Figure 1). The packets are moving from site A to site B (left to right), with the 
leading edge of both packets arriving at site B at the same time. The difference in link size (bandwidth) 
enables the high bandwidth link to complete delivery of the packet (loading and unloading) before the 
low bandwidth link. 
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CXO 7853b 



1 Site A 

2 T3 link (44.5 Mb/s) 

3 OC3 link (155 Mb/s) 

4 Site B 

Figure 1 Distance and I/O rate 



2q! TIP: For a visual reference, a nanosecond of fiber is approximately eight inches long. A bit is four inches 
'Y N long at 2 G b/ s and eight inches long at 1 G b/ s. Therefore, a nanosecond of fiber contains two bits at 2 

Gb/sand one bit at 1 G b/ s. At 100 M b/ s, a bit is 80 inches and at 10 Mb/s, a single bit is 

800 inches. 
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2 M a nua My estimating performance 



This chapter describes how to manually estimate the time needed to complete one 1/ 0 replication across 
a directly connected (or zero distance) intersite link. It also describes how to add the impact of distance to 
the calculation. 

Topics include: 

Using the formula without distance, page 11 
Adding distance, page 12 

Calculating performance for one outstanding write, page 12 

Using the formula without distance 

To calculate performance based on link bandwidth, use the following formula: 

Y = mX + b 

W here 

• y is the time (in milliseconds) to complete one outstanding replication 1/ 0 . 

• m is the slope of the line representing the link bandwidth. 

• x is the size of the transfer in kilobytes. 

• mx is the is the additional time required to complete a transfer (or data packet) larger than 512 bytes 
for a given link bandwidth. 

• b is the amount of time required to replicate a single 512 byte write (replication and conversion 
overhead). This is the y-intercept of the line representing the link bandwidth when x is zero. 

Use the values in Table 2 for the appropriate intersite link technology to complete this formula. 
NOTE: The values in Table 2 were derived experimentally. 



Table 2 Intersite link values 



Technology 


Bandwidth (Mb/ s) 


Intercept (b) 


Slope (m) 


2 Gb/s Fibre C hannel 


2000 


0.3416 


0.0268 


1 Gb/s Fibre C hannel 


1000 


0.3991 


0.0332 


1 GbE (Gigabit Ethernet) IP 


1000 


0.4130 


0.0338 


OC3 IP 


155.5 


0.3901 


0.0758 


E4 IP 


139.3 


0.3876 


0.0818 


100 Mb/s IP 


100 


0.3802 


0.1052 
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Table 2 Intersite link values (continued) 



Technology 


Bandwidth (Mb/ s) 


Intercept (b) 


Slope (m) 


T3 IP 


44 


0.3530 


0.2070 


E3 IP 


34.304 


0.3340 


0.2666 


10 Mb/sIP 


10 


0.2893 


0.8872 


El IP 


2.048 


1.1748 


4.4434 


Tl IP 


1.54 


1.5557 


5.9422 



Adding distance 

To obtain a more realistic estimate of performance, add the effects of distance, which is measured by the 
time required for the leading edge of the data to traverse the link. Estimate this time using a speed of 2 x 
10 8 meters per second for light in standard fiber-optic cable, which is equal to 5 microseconds per 
kilometer. For example, if the intersite link distance is 10,000 km (6,200 mi), the one-way time interval is 
50,000 microseconds, or 0.05 seconds. This is approximately 25 times the average rotational latency of 
a 15,000 rpm disk drive at 0.002 seconds. 

Therefore, the actual time to move data from the source array to the destination array is the round-trip 
distance, plus the time to load and unload the data for the link size. These trips consistof one small control 
packet and one large data packet and add 10 microseconds of latency (per kilometer) to the one-way 
intersite link distance to complete each replication write. Based on the distance between the two sites, this 
latency is added to the previously calculated time to complete a zero distance replication write. 



Calculating performance for one outstanding write 

During synchronous replication, an application performs one write and waits for that write to complete 
before issuing the next outstanding write. This section describes how to calculate performance for one 
outstanding write that contains a 32 KB data packet Based on the one-way network latency of 1 ms, the 
local and remote sites are 200 km (125 mi) apart 

Topics include: 

• Calculating replication time, page 13 

• Calculating time to load, page 13 

• Calculating time to complete, page 14 

• Determining maximum 1/ O s per second, page 15 
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Calculating replication time 

To measure the performance of this outstanding write: 

1. Use the values in Table 2 to calculate the loading time for a 32 KB data packet M ultiply the slope by 
the size of the transfer — m times x). 

2. Calculate and add the time ittakes for a round trip. If the intersite link distance is 200 km, the transfer 
(roundtrip) latency is calculated as follows (applies to all intersite link types): 

2 00 km x 5 ms/km x 2 = 2 ms 

3. Add the replication overhead from Table 2 (the Intercept column). 

The sum is the total time ittakes to replicate a 32 KB data packetfor a single outstanding write. 

Calculating time to load 

The time-to-load value is defined as the length of the data packet in time (the time to load and unload it). 
Using the intersite link bandwidth and slope values from Table 2, Table 3 shows the time-to-load value 
(step l)fora 32 KB data packet using different intersite link technologies^ 

Table 3 Time-to-load value 



Intersite link technology 


Slope 


Time to load (ms) 


2 Gb/s Fibre Channel 


0.0268 


0.86 


1 Gb/s Fibre C hannel 


0.0332 


1.06 


1 GbE (Gigabit Ethernet) IP 


0.0338 


1.08 


OC3 IP 


0.0758 


2.43 


E4 IP 


0.0818 


2.62 


100 Mb/s IP 


0.1052 


3.37 


T3 IP 


0.2070 


6.62 


E3 IP 


0.2666 


8.53 


10 Mb/s IP 


0.8872 


28.39 


El IP 


4.4434 


142.19 


Tl IP 


5.9422 


190.15 
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Calculating time to complete 

The time-to-complete value is defined as the time to complete one 1/ 0 replication across a given 
technology for a given packet size. It is the time for a host to send a synchronous replication write and 
receive an acknowledgement of completion. Table 4 show the time-to-complete value for a 32 KB data 
packet (the sum of step 2 and step 3) by adding transfer latency and replication overhead to the results in 
Table 3. 

fable 4 Time-to-complete value 



Intersite link technology 


Time to load 
(ms) 


+ Transfer latency 
(ms) 


+ Overhead 
(ms) 


= Time to complete 
1/ O (ms) 


2 Gb/s Fibre C hannel 


0.86 


2 


0.3416 


3.20 


1 Gb/s Fibre C hannel 


1.06 


2 


0.3991 


3.46 


1 GbE (Gigabit Ethernet) 
IP 


1.08 


2 


0.4130 


3.49 


OC3 IP 


2.43 


2 


0.3901 


4.82 


E4 IP 


2.62 


2 


0.3876 


5.01 


100 Mb/s IP 


3.37 


2 


0.3802 


5.75 


T3 IP 


6.62 


2 


0.3530 


8.98 


E3 IP 


8.53 


2 


0.3340 


10.87 


10 Mb/s IP 


28.39 


2 


0.2893 


30.68 


El IP 


142.19 


2 


1.1748 


145.36 


Tl IP 


190.15 


2 


1.5557 


193.71 
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Determining maximum 1/ 0 s per second 

You can use the time -to -complete value to determine the maximum number of synchronous replication 
writes that can be completed every second, assuming the next 1/ 0 starts immediately after the current 1/ 0 
completes. To do so, you must invert the the time to complete value into 1/ 0 s per second (10 PS). Table 5 
shows the results of the inversion for a 32 KB data packet which were calculated as follows: 

• Throughput is the packet size multiplied by the 1/ 0 rate (10 PS). 

• Bandwidth used is the peak bandwidth of the link, divided by the throughput and then multiplied by 
100. 



NOTE: Because most applications produce multiple asynchronous 1/ 0 s, Table 5 shows the minimum 
expected performance. 



Table 5 Bandwidth used for a 32 KB write 



Intersite link technology 


Approximate 10 PS 


Throughput (Mb/ s) 


Approximate single stream 
bandwidth used 


2 Gb/s Fibre C hannel 


312.58 


100.03 


5.0% 


1 Gb/s Fibre C hannel 


288.89 


92.45 


9.2% 


1 GbE (Gigabit Ethernet) 
IP 


286.16 


91.57 


9.2% 


OC3 IP 


207.65 


66.45 


42.9% 


E4 IP 


199.79 


63.93 


46.0% 


100 Mb/s IP 


174.02 


55.69 


55.7% 


T3 IP 


111.40 


35.65 


79.2% 


E3 IP 


92.04 


29.45 


86.6% 


10 M b/s IP 


32.59 


10.43 


100.0% 


El IP 


6.88 


2.20 


100.0% 


Tl IP 


5.16 


1.65 


100.0% 



Continuous Access EVA performance estimator user guide 15 



16 M anually estimating performance 



3 Using the Performance Estimator 



This chapter describes how to use the Performance Estimator, a M icrosoft Excel-based spreadsheet to 
perform the calculations described in "Manually estimating performance" on page 11. 

Topics include: 

• Accessing the spreadsheet page 17 

• Entering one-way latency, page 18 

• Entering data packet size, page 19 

• Reading the results, page 19 

• Comparing calculation results, page 21 

Accessing the spreadsheet 

The Performance Estimator spreadsheet is available from the following web site: 

http:// hl8006.wwwl.hp.com/ products/ storage/ software/ conaccesseva/ index.html 

Select Related information and then select HP StorageWorks Continuous Access EVA Performance 
Estimator to download the spreadsheet 

Figure 2 shows the default view of the spreadsheet when you open it W hen you enter values in the 
latency and size fields, the spreadsheet calculates the bandwidth for: 

• 2 Gb/ s fiber— Direct fiber or wave dimension multiplexing (W DM ) only. 

• 1 Gb/s fiber- Direct fiber over W DM only. 

• FC-SONETor FC-IP— See Table 2 on page 11 for a complete list of the intersite link technologies in this 
category. 

The 2 G b/ s and 1 G b/ s intersite links include long-distance direct fiber connections using longwave or 
very long distance G BIC / SFP, of either coarse- or dense-wave division multiplexing (C W DM or DW DM ). 

2f)l TIP: Place the mouse over any red triangle in the spreadsheet to view additional information for thatfield. 
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HP Storage Works Continuous Access EVA Replication Performance Estimator 
for a single outstanding write (synchronous or asynchronous} 


Average one-way latency: 




2.0 


ms 


400 km or 


249 miles 


Average data packet size: 




8] KB 


0-256 KB, or 1MB 










2 Gb/s fiber 




1 Gb/s liber 




FC-SONET/FC-IP 




FC-SONET/FC-IP 




Liak biadwidtk 


2000 


Mbls 


1000 


Mbls 


100.0 


Mbls 


5.0 


Mbls 


Picket load! 
■■load tiae: 


0.21 


ms 


0.27 


ms 




ms 


12.93 


ms 


Milliseconds per ^ 

HO: 


4.56 


ms 


4.66 


ms 




ms 


20.03 


ms 


IIOs per ftcoid: 


219.5 


IOPS 


214.4 


IOPS 


185.9 


IOPS 


49.9 


IOPS 




















Tkroigkpit (GbIL): 


6.32 


GBfh 


6.17 


GBIh 


5.35 


GBIh 


1.44 


GBIh 


Tkroigkpit (Mbls): 


17.56 


Mbls 


17.15 


Mbls 




Mbls 


3.99 


Mbls 


X baidwidtk 


0.88% 




1.72% 








79.90% 























Figure 2 Performance Estimator spreadsheet 



Entering one-way latency 

To begin using the spreadsheet enter the appropriate value in the Average one-way latency field. This is 
the distance between the source and destination arrays in milliseconds. Use the ping command to 
determine round-trip latency and then divide the result in half. You can also estimate latency using one of 
the following calculations: 

• For point-to-point networks, multiply the driving distance between sites by 1.5. 

• For routed networks, multiply the driving distance between sites by 2.25. 

W hen you enter a value and either press Enter or click outside the cell, the latency is translated into 
kilometers and miles. Figure 3 shows three examples. 

NOTE: The maximum value for this field is 100 mi lliseconds. 



Average one-way latency: 
Average one-way latency: 
Average one-way latency: 

Figure 3 Latency examples 
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Entering data packet size 

The other input is the average size of the application write being replicated to the destination array. The 
values for this field are 0-256 KB, or 1 M B for a full copy. You can estimate the size for the type of 
applications you are using. Some examples are: 

• 4 KB (such as M icrosoft Exchange) 

• 8 or 16 KB (small databases, such as M icrosoft SQ L) 

• 32 or 64 KB (large databases, such as 0 racle) 



IqI HP: If you have writes of multiple sizes (for multiple applications), complete an estimate for each write 
'V s size. 



Reading the results 

Using the information you enter, the Replication Performance Estimator spreadsheet calculates how latency 
and write size affect the bandwidth of the applicable intersite link technology. 

For all intersite link technologies listed (2 G b/ s fiber, 1 Gb/s fiber, and FC-SO N ET or FC-IP), the 
bandwidth results display as follows: 

• Link bandwidth— The amount of bandwidth (in M b/ s) 

• Packet load/ unload time— The length of time (in milliseconds) to move the data packet on to and off of 

the intersite link. It is based on the size of the outstanding write and the link bandwidth. 

• ms per 1/ 0 — The length of time to complete one synchronous 1/ 0 across the link. Starting with the 
intersite latency and packet transmit time formulas above. It is the summation of packettransmittime, 
plus twice the one-way latency, plus all conversion overhead. 

• 1/ 0 per second— The inverse of milliseconds per 1/ 0 . The maximum number of synchronous 1/ 0 per 
second for a single data stream. For synchronous 1/ 0 , the next 1/ 0 does not start until the one in 
progress has finished. The maximum rate can be achieved only when the next write starts immediately 
after completion of the previous write. In a real-world environment in which writes are generated in a 
pseudo-random fashion, the expected peak is approximately 70% of the theoretical peak. The average 
rate typically does not exceed 50% of the theoretical peak. Table 5 on page 15 shows the 1/ 0 s per 
second of various SAN technologies. 

• Throughput (GB/ h)— The transfer rate based on one hour of 1/ 0 s per second, multiplied by the data 
packet size. 

• Throughput (Mb/ s)— The transfer rate based on one second of 1/ 0 s per second multiplied by the data 
packet size. Add these numbers for each 1/ 0 stream to get estimated bandwidth requirements. 

• % bandwidth— The estimated bandwidth required for this single replication. It is based on the 
estimated throughput and the link bandwidth. Due to the mathematics of the model, the value may 
exceed 100%. 



Continuous Access EVA performance estimator user guide 19 



Figure 4 shows the results for latency of 5 milliseconds and data packet size of 32 KB. 





2 Gbj's fiber 




1 Gb.'s fiber 




FC - S ON ET/FC -IP FC-SON ET/FC -I P 




Liifc baidwidth 

Packet load! 
■ ■load ti-ae: 
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"° 

IfDs per secoad: 


2000 


Mbfs 

ms 
ms 
IOPS 


1000 


Mbls 


100.0 


Mbls 


5.0 


Mbls 


0.86 


1.06 


ms 
ms 
IOPS 


4.05 


ms 
ms 
IOPS 


51.73 


ms 
ms 
IOPS 


11.20 


11.46 




64.83 


89.3 


87.2 




15.4 




















Tkroagkpat (Gbli): 
Tkroagkpat (Mbfs): 

i biadwidtk 


10.29 


GBfh 
Mbfs 


10.05 


GBIh 
Mbls 




GBIh 
Mbls 


1.78 


GBIh 
Mbls 


28.57 


27.92 


22.19 


4.94 


1.43% 


2.79% 


22.19% 


98.73% 



Figure 4 Sample results 



Altering the link bandwidth 

You cannot change the link bandwidth fields for 2 G b/ s and 1 G b/ s fiber. You can, however, alter the 
link bandwidth for both FC-SO N ET/ FC-IP fields. This enables you to enter a value for the specific intersite 
link technologies you are using. The default values are 100 Mb/s(100 Mb/s Ethernet IP ISL) and 5 M b/ s 
(the minimum supported bandwidth for each ISL in a dual ISL connection). Figure 4 shows the results 
based on these default values. 

If you want to see the results for the 0 C 3 IP intersite link, enter 155.5 in the Link bandwidth field for the 
first FC-SO N ET/ FC-IP column. Figure 5 shows the results. 



NOTE: The commentfor the FC-SO N ET/ FC-IP column lists the value to enter for the corresponding ISL 





2 Gbi's fiber 




1 Gb/s fiber 




FC-SONET/FC-IP 


FC- SON ET/FC -IP 
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Iffls per fecoad: 


2000 


Mbfs 

ms 
ms 
IOPS 


1000 
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Mbls 
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Mbfs 

ms 
ms 
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0.86 


1.06 


ms 
ms 
IOPS 


2.79 


I 

ms 
ms 
IOPS 


51.73 


11.20 


11.46 
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87.2 


76 0 


15.4 




















Tkroagkpat (Gblk): 
Tkroagkpat (Mbfs]: 

* biadwidtk 


10.29 


GBfh 
Mbfs 


10.05 


GBIh 
Mbls 




GBIh 
Mbls 


1.78 


GBfh 
Mbfs 


28.57 


27.92 




4.94 


1.43% 


2.79% 


15.63% 


98.73% 



Figure 5 Sample results for 0 C3 IP intersite link 

As another example, you want to see the bandwidth results for the T3 IP intersite link. Change the Link 
bandwidth value for the first FC-SO N ET/ FC-IP column to the default (100) and enter 44 in the second 
FC-SO N ET/ FC-IP column. Figure 6 shows the results. 
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r^f NOTE: HP recommends that you begin with the default values for the FC-SO N ET/ FC-IP fields. As your 
\=-J analysis proceeds, adjust one value to compare with the default value. 
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Figure 6 Sample results for T3 IP intersite link 



Comparing calculation results 

Figure 7 shows that latency of 1 millisecond and data packet size of 32 KB consumes approximately 50% 
of a 100 M b/ s FC-IP intersite link. The average load on any link must not exceed 40% of its rated 
capacity, and the peak loading must not exceed 45% of rated capacity. This limitation allows 1/ 0 from a 
failed link or fabric to run on the nonfailed fabric or link without causing additional failures by 
overloading the active fabric. 

The result of the manual calculation for the same latency and data packet size is 55.77% (see Table 5 on 
page 15). 
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r2^> NOTE: There may be a difference between the results you generate manually and those you generate 
L_J using the spreadsheet The Performance Estimator spreadsheet uses a mathematical model based on the IP 

link bandwidth to estimate the slope and intercept of the line that is used in the first two rows of 

calculations. 
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Figure 7 Results for a 32 KB write 
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4 Determining maximum bandwidth 



The previous chapters showed results for one outstanding write. However, most applications do not issue 
one write and waitfor itto complete before immediately sending the nextone. Instead, multiple writes are 
replicated simultaneously (asynchronous replication), which requires that you determine the effect of 
multiple writes from a single application. This chapter describes how to use those calculations for multiple 
writes. 

Topics include: 

• M aximum number of messages, page 23 

• Peak write rate, page 24 

• Effects of multiple writes, page 27 

• Actual 1/ 0 s per second, page 28 



M aximum number of messages 

A high bandwidth link can accommodate more data than a low bandwidth link (Figure 8). The maximum 
capacity is called the bandwidth-latency product To calculate the maximum number of simultaneous 
messages allowed in the communications link: 

1. M ultiply the net performance of the communications link (the total bits per second, minus overhead) by 
the one-way intersite latency (in seconds). 

2. Convert the result into bytes (Fibre Channel uses 10 bits per byte) and divide by the average message 
size (in bytes). 

In calculating this number, it is assumed thatthe application can issue 1/ 0 as soon as space is available 
on the link, thus streaming the data from the host to the source array. This data is then replicated to the 
destination EVA with acknowledgment back to the host It is also assumed that because there is only one 
source, data is written regularly and consistently. These assumptions are necessary to understand peak 
performance. 
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Figure 8 M ultiple writes for a single application 



Peak write rate 

In Figure 8, the parallelograms represent multiple write data streams from the same application. Each data 
stream contains data packets of equal size (same number of bytes). The low bandwidth link can 
accommodate three packets, but the high bandwidth link can accommodate nine packets. Itis assumed 
that each new packet immediately follows the preceding packet regardless of link bandwidth. 

To determine the approximate peak write rate for an application: 

1. Invert the single write percentage of bandwidth used (Table 5 on page 15) and divide by 100. The 
result is the maximum number of data streams that can be supported by that link for that distance. 

2. M ultiply the result by the single write rate. 

Limitations 

Between any two array controllers in a replication relationship, a bidirectional tunnel is created to 
transport the data (full copies and outstanding writes). A tunnel is a path that exists within the intersite link 
between two controllers and supports independent data transfers. W ith two controllers in each array, the 
four controllers create four tunnels (two active and two standby). A DR group belongs to only one tunnel, 
but one tunnel can support multiple DR groups. 

The number of replication relationships required determines the total number of tunnels that are created 
(four for each relationship). Further, the number of tunnels determines the minimum required bandwidth for 
a specific communications link. For example, you want to consolidate two sources into one destination 
supporting eight tunnels (four active and four standby). Add the throughput requirements for each active 
tunnel using a given intersite link technology. This sum (the total throughput requirements for all tunnels) 
determines the maximum required bandwidth for that communications link. 

The following factors can limit the expected maximum number of writes in the link: 

• Raw capability of the host; FCA, and the source array controller 

For example, to ensure write order across all members of a DR group, the members are restricted to 
using the same FCA port and array controller and Secure Path's load balancing is turned off. 
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• N umber of buffer-to-buffer credits allowed between two Fibre Channel devices 

For each replication relationship, buffers process the replication 1/ 0 s. The number of buffers depends 
on the intersite latency. For example, if the intersite latency is small, not all buffers are used. As the 
intersite latency increases, the number of buffers used will increase, which may become a performance 
limitation. 

Tables 6, 7, and 8 list the buffer-to-buffer credits for B-series, C-series, and M -series switches, 
respectively. This limit becomes the bottleneck on a long-distance direct Fibre C hannel connection with 
very long-distance gigabit interface converters (G BICs) or a wavelength division multiplexing (W DM ) 
solution. It is not usually seen in Continuous Access EVA over IP configurations because the credit is 
returned by the IP gateway to the sending switch. 

• Maximum number of outstanding writes the array controller can support 

Using Virtual Controller Software version 3.00 or 3.01, the tunnel within the array controller allows up 
to 31 (8 KB) outstanding writes per controller port If the bandwidth latency product of the link can 
support more than 31 writes (such as in high speed, very long distance Continuous Access EVA over IP 
configurations), the maximum number of 8 KB or smaller outstanding writes in the link will be 31 for 
each array controller sharing the link. 

Using Virtual Controller Software version 3.02 or later, up to 62 outstanding writes are allowed for 
each controller port In a worst-case situation in which one port of each controller uses the same link, 
the combined limit is 124 (62 for each port) outstanding writes, subject to other limits. 

• Impact of the initial full copy process on the intersite link 

In all 3.x versions of Virtual Controller Software, the full copy process allocates eight 128 KB data 
buffers. These buffers are only used during the initial full copy (a sequential read/ write process in 
which a complete copy of the source array' s content is made on the destination array) or after the 
write history log becomes full. W hen the log is full and the DR group members are marked for full 
copy, either a full copy or a fast resynchronization occurs. Fast resynchronization is the process of only 
moving those blocks of data that changed instead of the entire contents of the DR group members. This 
process uses random reads to find the data that must be copied to the destination array. 

Replication consumes as much as 492 KB (62 x 8 KB), but the full copy consumes as much as 1 M B 
(8x128 KB). Combined, these two processes may overwhelm an intersite link if it does not have 
enough bandwidth. 
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Table 6 FC buffer-to-buffer credits for B-series switches 



Switch family 


Default credits 


Credits available with 
Extended Fabric License 


3xxx at 1 G b/ s 


16 


60 


4xxx at 1 G b/ s 


27 


60 


3xxx at 2 G b/ s 


27 


64 


4xxx at 2 Gb/s 


27 


64 


Table 7 FC buffer-to-buffer credits for C -series switches 


Switch family 


Default credits 


Maximum available credits 


All 16-port modules, Fx mode 


16 


255 


E or TE modes 


255 


255 


All 32-port modules 


12 


N/A 


Table 8 FC buffer-to-buffer credits for M -series switches 


Switch family 


Default credits 


Maximum available credits 


3 xxx 


60 


60 


4 xxx 


162 


162 


6 xxx 


60 


60 
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Effects of multiple writes 

Figure 9 represents multiple writes from multiple applications. The difference between Figure 9 and the 
single application in Figure 8 is that the space between the writes is wider for multiple applications. The 
wider space reduces the number of writes in the intersite link and also reduces the maximum utilization 
rate of that link. Any differences in the number of writes in the link are based on how the bandwidth is 
shared between the applications. 



2q1 TIP: If you are using multiple high-performance applications, H P recommends that you add Continuous 
'V* Access EVA storage arrays to maintain performance. 



In real-world application environments, it is not possible for all writes to arrive at precisely the expected 
times. Independent applications do not coordinate with each other when sending reads or writes to the 
source array. This lack of coordination creates space between any two writes on the intersite link. 
M athematical queue theory suggests thatthe expected peak utilization is approximately 70% and the 
expected average is approximately 50%. In addition, there must be sufficient bandwidth on both fabrics 
for all traffic, if one fabric fails. If there is an even split between the two fabrics, the 50% rate becomes 
25%. 
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Figure 9 M ultiple writes for multiple applications 
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To determine the effect of multiple writes on the intersite link, consider the following example. There are 
two applications— one performs 32 KB writes and one performs 2 KB writes. Table 5 on page 15 shows 
the bandwidth used for a 32 KB write. Table 9, using the same formula as in Table 5, shows the 
bandwidth used for a 2 KB write. 



Table 9 Bandwidth used for a 2 KB write 



Intersite link technology 


Approximate 10 PS 


Throughput (M b/ s) 


Approximate single 
stream bandwidth used 


2 Gb/s Fibre C hannel 


417.50 


8.35 


0.4% 


1 Gb/s Fibre C hannel 


405.60 


8.11 


0.8% 


1 GbE (Gigabit Ethernet) 
IP 


403.13 


8.06 


0.8% 


OC3 IP 


393.44 


7.87 


5.1% 


E4 IP 


391.97 


7.84 


5.6% 


100 M b/ s IP 


386.01 


7.72 


7.7% 


T3 IP 


361.40 


7.23 


16.1% 


E3 IP 


348.77 


6.98 


20.5% 


10 Mb/sIP 


246.08 


4.92 


49.2% 


El IP 


82.91 


1.66 


82.9% 


Tl IP 


64.77 


1.30 


86.4% 



Actual 1/ O s per second 

Determine the actual number of 10 PS expected from each application by scaling the single write rate 
(either up or down) to the expected rate, and then scaling the bandwidth needed by the same number. If 
the total required bandwidth exceeds 25%, you cannot expect that link to support the replication 
requirements. For example, one application read and write produces transactions at 3 times the 32 KB 
single stream rates. The other read and write produces transactions at 4 times the 2 KB single stream 
rates. 
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Table 10 shows thatonly the 2 G b/ s link can support the replication requirements. If high-speed links are 
not available, you can use multiple low-speed links to obtain the required 25% of available bandwidth. 



table 10 Total bandwidth required for multiple 1/ 0 streams 



Intersite link 
technology 


32 KB 
10 PS 

(x3) 


32 KB 
throughput 
(M bl s) 


Bandwidth 
used 


2 KB 
10 PS 

(x4) 


2 KB 

throughput 
(M bl s) 


Bandwidth 
used 


Total 

bandwidth 
required 


2 Gb/s Fibre 
C ha nnel 


938 


300 


15% 


1670 


33 


2% 


17% 


1 Gb/s Fibre 
C hannel 


867 


277 


28% 


1623 


32 


3% 


31% 


1 G bE (G igabit 
Ethernet) IP 


859 


274 


27% 


1613 


32 


3% 


31% 


OC3 IP 


623 


199 


129% 


1574 


31 


20% 


149% 


E4 IP 


599 


192 


138% 


1568 


31 


23% 


161% 


100 M b/ s IP 


522 


167 


167% 


1544 


31 


31% 


198% 


T3 IP 


334 


107 


238% 


1446 


29 


64% 


302.% 


E3 IP 


276 


88 


260% 


1395 


28 


82% 


342% 


10 Mb/sIP 


98 


31 


313% 


984 


20 


197% 


510% 


El IP 


21 


6.6 


330% 


332 


6.6 


332% 


662% 


Tl IP 


15 


5 


330% 


259 


5.2 


345% 


676% 



A CAUTION: Design a solution thatwill notoverload a controller or intersite link during normal operations 
to prevent significant loss of performance during failure conditions. This is especially true when using 
asynchronous replication and during the initial full copy or normalization process that occurs when you 
create a new DR group. 
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